A subjective measure of web search quality

In an Internet search, the user uses a query language to describe the nature of documents, and, in response, a search engine locates the documents that ‘‘best match’’ the description. A number of search engines are available to Internet users today and more are likely to appear in the future. These systems differ from one another in the indexing technique they use to construct the repository of documents and the search algorithm they employ to generate the response. As a result, the results for the same query from different search engines vary considerably. In this work, our aim is to outline a procedure for assessing the quality of search results obtained through several popular search engines. This procedure would enable us to compare the performance of the popular search engines like AltaVista, DirectHit, Excite, Google, HotBot, Lycos and Yahoo, etc. We measure the ‘‘satisfaction’’ a user gets when presented with the search results. We watch the actions of the user on the search results presented before him in response to his query, and infer the feedback of the user therefrom. The implicit ranking thus provided by the user is compared with the original ranking given by the search engine. The correlation coefficient thus obtained is averaged for a set of queries. We show our results pertaining to 7 public search engines and 15 ad hoc queries. Our emphasis is more to demonstrate the procedure of quality measurement than to carry out the actual performance measurement of these search engines. 2004 Elsevier Inc. All rights reserved. 0020-0255/$ see front matter 2004 Elsevier Inc. All rights reserved. doi:10.1016/j.ins.2004.07.003 * Tel.: +91 11 2659 6120; fax: +91 11 2658 1274. E-mail address: [email protected] 366 M.M. Sufyan Beg / Information Sciences 169 (2005) 365–381